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10 BACKGROUND OP THE TNVENTTnN 

Field of the Invention 

This invention relates generally to modeling language for 3D graphics and, more 
particularly, to temporal manipulation of media assets. 



Description of the Related Prior Art 



Conventional modeling languages for real-time 3D scene rendering have traditionally 
focussed on aspects of scene stiuctuie. geometry, appearance, and, to some degree, animation, 
and interactivity. This focus has been driven by the following two factors. First, 3D 
computer graphics applications have been geared toward user-driven experiences and. Uius, 
tend to be structiired around a rendered response to events. Second, the majority of tfiese 
2 0 applications take a "render it as fast as you can" approach to scene updates, with little respect 
paid to fidelity of tiie time base. Conventional modeling languages fail to provide the 
accuracy of temporal relationship between two media assets. For example, if a video asset 
and an audio asset are to start at the same time, tiiis can be achieved by prescribing start time 
for each asset independent of other assets. This allows Uie start times to be slightiy different. 
25 It is desirable that th start time for each asset be controlled by the same field, thereby 



Puent/S0N34S8.01 



resulting in accurate synchronization of the assets. Media assets include audio media, video 
media, animations, audio-visual media, images or events. 

As full motion video and high fidelity audio are integrated into a scene rendering mix 
it is desirable to deliver high quahty television-like viewing experiences while supporting 
viewer interactivity. It is desirable to provide a passive viewing experience that is more 
television-like and not a web page-like viewing experience. 

In a declarative markup language, the semantics required to attain the desired outcome 
are implicit in the description of the outcome. It is not necessary to provide a separate 
procedure (i.e., write a script) to get the desired outcome. One example of a declarative 
language is HypeiText Markup Language (HTML). 

Various approaches to scoring animation and playback have previously been 
developed in other computer-based media, including Macromedia Director and the W3C's 
Synchronized Multimedia Integration Language (SMIL). However, these existing scoring 
systems do not allow for declarative composition of a real-time scene wherein the 
independent scores are dynamically composed and decomposed hierarchically, structuring 
time in manner akin to tiie spatial scene graph. For example structuring blocks of time to be 
next to each other or stnicturing block of time to be parallel (synchronized) witfi each otiier. 
The conventional scoring systems do not allow variable rate and direction of score evaluation 
to be done declaratively. and neither do they allow declarative implementation of a modular 
computation strategy based upon a normalized "fraction done" output, suitable for rapid 
assembly and reuse of behavioral animation. 

SUMMARY OF THE INVRNTTHM 

A system and method for declarative markup that allows temporal manipulation of 
media assets is presented. The media assets can be audio media, video media, animations, 
audio-visual media, images or events. Using tiie present invention a media sequence can be 
formed by playing more than one medium in series, in parallel or in any other temporal 
combination wherein a medium is cued to another medium. A media sequence created using 
the present invention can become part of a new media sequence, and die rate of playing the 
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media sequence can be controlled by fields associated with the new media sequence. Also, 
using present invention, a media sequence can be cued to start playing at a fixed time before 
the end of a first media sequence, and in this instance the length of the first media sequence 
can be varied while still maintaining the fixed time from the end of the first media sequence. 

5 BRIEF DESCRIPTIQN OF THE PR A WfNn 

Fig. lA shows the basic architecture of Blendo. 

Fig. IB is a flow diagram illustrating flow of content through Blendo engine. 

Fig. 2A shows time relationship between media sequences in a score. 

Fig. 2B illustrates synchronization of the media sequence of Fig. 2 requiring pre- 

10 loading 

Fig. 3 shows time relationships between various constituent media of an interactive 
presentation. 

DHTAn.F.r> HRgrPTPTrnM 
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Blendo is an exemplary embodiment of the present invention that allows temporal 
manipulation of media assets including control of animation and visible imagery, and cueing 
of audio media, video media, animation and event data to a media asset that is being played. 
Fig. lA shows basic Blendo architecture. A comprehensive description of Blendo can be 
found in Apehdix A. At the core of the Blendo architecture is a Core Runtime module 10 
(Core hereafter) which presents various Application Programmer Interface (API hereafter) 
elements and the object model to a set of objects present in system 1 1. During normal 
operation, a file is parsed by parser 14 into a raw scene graph 16 and passed on to Core 10. 
where its objects are instantiated and a runtime scene graph is built. The objects can be built- 
in objects 18. author defined objects 20, native objects 24. or the like. The objects use a set of 
available managers 26 to obtain platform services 32. These platform services 32 include 
event handling, loading of assets, playing of media, and the like. The objects use rendering 
layer 28 to compose intermediate or final images for display. A page integration component 
30 is used to interface Blendo to an external environment, such as an HTML or XML page. 

Patent /$0N34S8.OI .3. 



Blendo contains . system object with references to the set of managen, 26. Each 

n»n.gcr26p,ovid«U,es« Of APIs to cont^I some aspectofsystemll. An even, manager 
26D provides access to incoming system events originated by user input or envi™,me„,3i 
events. A load manager 26C facilitates the loading of Blendo files and native node 
5 .mplementations. A media manager 26E provides the ability to load, control and play audio 
.mage and video media a^ts. A render manager 260 allows the cteaUon and management of 
objectsusedto-enderscenes. A scene manager 26A conttols the scene graph. AsuL 
manager 26F allows the creaUon and managen^t of surfaces onto which scene elements and 
otherassetsmaybecomp^ited. A thread manager 26B gives authors the ability to spawn 
10 and conuol threads and to communicate between them. 

Fig. IB illustrates in a flow diagram, .conceptual description of the flow of content 
through a Blendo engine. In block 50. a presentation begins with a soutce which includes a 
file or stream 34 (Kg. lA) of cont«,t being brought into patser 14 (Kg. lA). The soutce 
could be in a naUv. VRMI^Iilce textual format, a nattve binary fbtmat. an XML based format 
orthehke. I^teof thefonnatoftheso.«ce,inblock55,thesourceiseonver,edi„.o ' 
raw scene graph 16 (Kg. lA). The raw scene gt^h 16 can reptesent the nodes, fields and 
other objects in the content, as weU as field initialization values. It also can contain a 
desoiption of object prototypes, external prototype teferences in the stteam 34. and route 
Statements. 

nie top level of raw scene graph 16 include nodes, top level fields and funcUons 
prototypes and mutes contained in the file. Blendo allows fields and functions at the top' level 
in addition to traditional elements. These are used to provide an interface to an external 
environment, such as an HIML page. They also provide the object interface when a stream 
34 IS used as the contents of an external prototype. 

25 Each raw node includes a list of the fields initialized within its context. Each raw field 

entry includes the name, type (if given) and data value(s) for that field. Each data value 
includes a number, a string, a raw node, and/or a raw field that can represent an explicitly 
typed field value. 
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m block 60. the prototypes are extracted from the top level of raw scene graph 16 (Fig 
lA) and used to populate the database of object prototypes accessible by this scene. 

The raw scene graph 16 is then sent through a build traversal. During this traversal 
each object is built (block 65). using the database of object prototypes. 

In block 70. the routes in stream 34 are established. Subsequently, in block 75 each 
field in the scene is initialized. This is done by sending initial events to non-default fillds of 
objects. Sincethescenegraphstiuctuteisachieved through the useofnode fields block 75 
also constructs the scene hierarchy as well. Events are fired using in-order t,.vet.al The first 
node encounteted enumerates fields in the node. If a field is a node, that node is traversed 
first. 

As a result the nodes in that particular branch of the tree are initialized Then, an 
event is sent to that node field with the initial value for the node field. 

After a given node has had its fields initialized, the author is allowed to add 
initialization logic (block 80) to prototyped objects to ensure that the node is fully initialized 
at call time. The blocks described above produce a root scene. In block 85 the scene is 
deKveredtothescenemanager26A(F,g. 1 A) created for the scene. In block 90. the scene 
manager 26A is used to render and perform behavioral processing either impliciUy or under 
author control. 

A scene rendered by the scene manager 26A can be constructed using objects from the 
Blendo object hierarchy. Appendix B shows the object hierarchy and provides a detailed 
description of the objects in Blendo. Objects may derive some of their functionality from 
their parent objects, and subsequenUy extend or modify their functionality. At the base of the 
hierarchy is the Object The two main classes of objects derived from the Object am a Node 
and a Reld. Nodes contain, among other things, a tender method, which gets called as part of 
theiendertraversal. The data properties of nodes arc called fields. Among the Blendo object 
hierarchy is a class of objects called Timing Objects, which are described in detail below. 
The following code portions are for exemplary purposes. It should be noted that the line 
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numbers in each code portion merely represent the line numbers for that particular code 
portion and do not represent the line numbers in the original source code. 

TIMING OBJECTS 

Timing objects include a TimeBase node. This is included as a Held of a timed node 
5 and supplies a common set of timing semantics to the media. Through node instancing the 
TimeBase node can be used for a number of related media nodes, ensuring temporal 
synchronization. Blendo also provides a set of nodes including Score which is utilized for 
sequencing media events. The Score is a timed node and derives its timing from a TimeBase. 
The Score includes a list of Cue nodes, which emit events at the time specified. Various 
1 0 timing objects, including Score, are described below. 

TimedNode 

The following code portion illustrates the TimeNode node. A description of the 
functions in the node follows thereafter. 

1) TimedNode : ChildNode { 
15 2) field TimeBaseNode timeBase NULL 

3) function Time getDurationQ 

4) fiinction void updateStartTime(Time now. Time mediaTime, Hoat rate) 

5) function void updateStopTime(Time now. Time mediaTime, Float rate) 

6) function void updateMediaTime(Time now. Time mediaTime. Hoat rate) 



20 } 



This object is the parent of all nodes controlled by a TimeBaseNode. 



In line 2 of the code portion, the TimeBase field contains the controlling 
TimeBaseNode. which makes the appropriate function calls listed below when the time base 
starts, stops or advances. 

25 In line 3, the getDuration function returns the duration of the TimedNode. If 

unavailable, a value of -1 is returned. This function is typically overridden by derived 
objects. 
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Lme4 lists the updateStaitTime function. When called, this function starts advancing 
.ts events or tontrolled media, with a starting offset specified by the mediaTime argument 
The updateStartTime function is typically overridden by derived objects. 

Line 5 lists the updateStopTime function, which when called, stops advancing i 
events or controlled media. This function is typically overridden by derived object 

I"«-6.theupdateMediaTimefunctioniscalledwhenevermediaTimeisupdatedby 
the TrmeBaseNode. The updateMediaTime function is used by derived objects to exert 
further control over their media or send additional events. 

IntervalSensor 

The foUowing code portion illustrates the IntervalSensor node. A description of the 
fields in the node follows thereafter. 

1) IntervalSensor : TimedNode { 

2) field Time cyclelntervall 

3) field Float fixation 0 

4) field Float time 0 



} 



The IntervalSensor node generates events as time 



passes. 



IntervalSensor node can be used for many purposes including but not limited to: 

• driving continuous simulations and animations; 

• controlling periodic activities one per minute); 

• initiating single occurrence events such as an alarm clock. 

The IntervalSensor node sends initial fraction and time events when its 
updateStartTimeO function is called. This node also sends a fraction and time event every 
time updateMediaTimeO is called. Finally, final fraction and time events are sent when the 
updateStopTimeO function is called. 
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'"K«2oftf,.codeporto„.U,ecycleI„,=„aineldiss«by,h.au,hor,otoern,i„e 
tt» icnph of Bm., n»a«^ i„ «co„ds. i, uk« for f™:,io„ ,„ go th>m 0 ,o 1 . TOs valu. is 
iBIunied when the gelDuiaionO funcdon is called. 

Une 3 Hsu fte f««io„ field, which generates events whenever the TimeBaseNode is 
running using equation (1) below: 

fraction = max(min(mediaTime/cycleInterval 1) 0) 

' ' Eqn. (i) 

Line 4 lists the time field, which generates events whenever the TimeBaseNode is 
ninmng. The value of the time field is the current wall clock time. 

Score 

The following code portion illustrates the Score node. A description of the field in the 
node follows thereafter, 

1) Score : TimedNode { 

2) field MF CueNode cue f 1 

} 

This object calls each entry in the cue field for every updateStaitTimeQ 
updateMediaTimeO and updateStopTimeQ call received. Calls to each cue ent^ returns the 
curxenUy accumulated relative time. This value is passed to subsequent cue entries to allow 
relative offsets between cue entries to be computed. 

In line 2 of the code portion, the cuefield holds the list of CueNode entries to be called 
wifli the passage of mediaHme. 

TimeBaseNftdft 

The following code portion illustrates the TimeBaseNode node. A description of the 
fields and functions in the node follows. 

1) TimeBaseNode : Node { 

2) field Time mediaTime 0 

3) function void evaluate(Time time) 

4) function void addCIientCTimedNode node) 

5) function void removcCIientCTimedNode node) 
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} 



6) function Int32 getNumCHents() 

7) function TimedNode getClientant32 index) 

This object is the parent of all nodes generating mediaTime. 



Line 2 of the code portion lists the mediaTime field, which generates an event 
Whenever mediaTime advances. MediaTime field is typically controlled by derived objects. 

Line 3 lists the evaluate function, which is called by the scene manager when time 
advances if this TimeBaseNode has registered interest in receiving time events. 

One 4 lists addClient function, which is called by each TimedNode when this 
TimeBaseNode is set in their timeBase field. When mediaTime starts, advances or stops 

eachclientin the list is called. If the passed node is alreadyaclien, this function perfonLs no 
Operations. 

Line 5 lists the removeClient function, which is called by each TimedNode when this 
TimeBaseNode is no longer set in their timeBase field. If the passed node is not in the client 
hst, this function performs no operations. 

One 6 lists the getNumCIients function, which returns the number of clients currently 
in the client list 

Line 7 lists the getClient function, which returns the client at the passed index. If the 
index is out of range, a NULL value is retumed 

TimeBase 

The following code portion illustrates the TimeBase node. A description of the fields 
in the node follows thereafter. 

1) TimeBase : TimeBaseNode { 

2) field Bool loop false 

3) field Time startTime 0 

4) field Time playTime 0 

5) field Time stopTime 0 

6) field Time mediaStartTime 0 

7) field Time mediaStopTime 0 
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8) field Float rate 1 

9) field Time duration 0 

10) field Bool enabled tnie 

11) field Bool isActive false 

} 

This object controls the advancement of mediaTime. TimeBase can start, stop and 
resume this value, as well as make mediaTime loop continuously. Time Base allows 
mediaTime to be played over a subset of its range. 

In line 2 of the code portion, the loop field controls whether or not mediaTime repeats 
Its advancement when mediaTime reaches the end of its travel. 

In line 3. the startTime field controls when mediaTime starts advancing. When 
startTime. which is in units of wall clock time, is reached the TimeBase begins ninning This 
is true as long as stopTime is less than startTime. When this occurs mediaTime is set to the 
value of mediaStartTime if rate is greater than or equal to 0. If mediaStartTime is out of 
range (see mediaStartTime for a description of its valid range), mediaTime is set to 0 If the 
rate is less than 0. mediaTime is set to mediaStopTime. If mediaStopTime is out of range 
mediaTime is set to duration. The TimeBase continues to mn until stopTime is inched or 
mediaStopTime is reached (mediaStartTime if rate is less than 0). If a startTime event is 
received while the TimeBase is running, it is ignoiwL 

In line 4 and 5. the playTime field behaves identically to startTime except that 
mediaTime is not reset upon activation. The playTime field allows mediaTime to continue 
advancing after the TimeBase is stopped with stopTime. If boti. playTime and startTime have 
the same value. startTime takes precedence. If a playTime event is received while the 

TimeBase is running, the event is ignored The stopTime field controls when the TimeBase 
Stops. 

In line 6. the mediaStartTime field sets the start of the subrange of the media duration 
over which mediaTime shall run. Therangeof mediaStartTime is from zero to the end of the 
duration (O..duration). If the value of mediaStartTime field is out of range. 0 is used in its 
place. 



Piaient/S0N34S8.01 



.10- 



In line 7. *e mediaStopTime fleld s«s U« «,<! of tt,e subrange of .he media du™io„ 
o«>r which mediaTin» shI. „„. The ange of mediaScopTime is from zero ,„ ,he end of ,he 
duration (O..dur.tionl. If .he value of mediaS.opTime is ou, of n,„ge. du^ion is u«d in i« 

place. 

In line 8, ,he rate field allows medlaTim. ,o ■«„ a, a raK other than one second per 
seeond of wall clock «n,e. H,e r.,. p^vided in fte «te field is used as a„ ,„s.a„,a„=ous rare 
When evaluate is call«l a,e elapsed an» since fte las, callis muUipHed by ra<e and tf,e ,«ul. 
is added to the current mediaTime. 

In line 9. d,e duration field gen«a<es an event when the duration of all clients of this 
TtmeBas. have detemUned their dutation. n„= y>i^ of the dur«ion field is the same as the 
client witfi the longest duration. 

In line 10, the enabled fieldenables d,eTimeBase. When enabled goes false, isActive 
goes false if it was true and mediaTinM stops «Ivancing. While false. startTime and 
playiime are ignored. When enabled field goes tnre. startTime and playrime are evaluated to 
detennineif thertnKBase should begin runmng. If so. m. behavior as descdbed in stanTime 
or playTime is perfonned. 

Line 1 1 lists the isActive field, which genemtes a true event when the TimeBase 
becomes active and a false event when the timeBase becomes inactive. 

CueNode 

ThefollowingcodesnippetillustiatestheCueNodenode. A description of the fields 
in the node follows thereafter. 

1) CueNode : Node { 

2) field Float offset -1 

3) field Float delay 0 

4) field Bool enabled true 

5) field Int32 direction 0 

6) function void updateStartTimeCTime now. Time mediaTime. Float rate) 
8 1 !cn";° "IKlateStopTimeaime now. Time mediaTime Float rate 
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10) function void fiieCTfme now. Time mediaTime) 

Tliis object is the parent for all objects in the Score's cue list. 

hH„e2of,hecodepo«ion.,heoffse.fleldes«bl,shesaO«,a«veoffse,to„, 
tKe beg,nn,ng of the sequence. For instance, a value of 5 win Are ,he CueNode when d,e 
incoming mediaTime reaches a value of 5. 

In line 3. the delay field establishes a relative delay before the CueNode fires 
l^offset.savalueotherthan-l(the default), this delay is measui^fro^ 

the delay is n^un^d from the end of thept.viousa.eNodeorfh>mOif this i^ 
CueNode. For instance, if offset has a value of 5 and delay has a value of 2. this node will 
firewhenmediaTimereachesV. If offset has a value of- 1 and delay has a value of 2 this 
node will fire 2 seconds after the pievious CueNode ends. 

In line 4. if the enabled field is false, the CueNode is disabled. The CueNode behaves 
as though offset and delay were their default values and it does not fire events. If it is true, 
the CueNode behaves normally. 

In line 5, the direction field controls how this node fires relative to the direction of 
travel of mediaTime. If this field is 0, this node fires when this node's offset and/or delay are 
reached, whether mediaTime is increasing (rate greater than zerx» or decreasing (me less than 
zero). If direction field is less than zero, this node fires only if its offset and/or delay are 
leached when mediaTime is decreasing. If direction field is greater than zero, this node fires 
only if this node's offset and/or delay are reached when mediaTime is increasing. 

Une 6 lists the updateStartTime function, which is called when the parent Score 
receives an updateStartTimeQ function call. Each CueNode is called in sequence. 

Line 7 lists the updateStopTime function, which is called when the parent Score 
receives an updateStopTimeQ function call. Each CueNode is called in sequence. 

Une 8 lists the evaluate function, which is called when the parent Score r^eives an 
updateMediaTimeO function call. Each CueNode is called in sequence and must return its 
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accumulated time. For instance, if offset is 5 and delay is 2. the CueNode would return a 
value of 7. If offset is -I and delay is 2. the CueNode would return a value of the incoming 
accumulated time plus 2. This is the default behavior. Some CueNodes (such as IntervalCue) 
have a well defined duration as well as a firing time. 

In line 9. the getAccumulaterfTime function returns the accumulated time using the 
same calculation as in the evaluate() function. 

Line 10 lists the fire function, which is called from the default evaluateQ function 
when the CueNode reaches its firing time. The fire function is intended to be overridden by 
the specific derived objects to perfoim the appropriate action. 

MediaCue 

The following code portion illustrates the MediaCue node. A description of the fields 
in the node follows thereafter. 

1) MediaCue : CueNode TimeBaseNode { 

2) field Time mediaStartTime 0 

3) field Time mediaStopTime 0 

4) field Time duration 0 

5) field Bool isActive false 

} 

This object controls the advancement of mediaTime when this CueNode is active 
MediaCue allows mediaTime to be played ov« a subset of its mige. MediaCue is active 

from the time deteiinined by the offset and/or delay field foralength of time detennined by 
mediaStopTime minus mediaStartTime. The value MediaCue returns from 

getAccumulatedTimeO is the value computed by adding the default function to the 
mediaStopTime and subtracting the mediaStartTime. This node generates mediaTime while 
active, which Is computed by subtracting ti.e firing time plus mediaStartTime from the 
incoming mediaTime. MediaCue therefore advances mediaTime at the same rate as the 
incoming mediaTime. 

In line 2 of die code portion, the mediaStartTime field sets the start of the subrange of 
the media duration over which mediaTime runs. TTie range of mediaStartTime is from zero to 
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«..«.dof<h.duration(0..*.ra.ion). "the v.Iu.ofm«iiaS«„T,™ field i.ou. of ran- Oi. 
utilized in its place. ^ 

In line 3. .he n^iaStopTime field s«s fte end of U>e subrange of U,e media duration 
over ,^ich mediaTime n„s. The range of mediaStopT.me is from ^ ,o the end of the 
du^aon (0.,du«i«„. ,f tt,e value of mediaStopTime field is out of range, duration is utiHzed 
in Its place. 

In line 4. the duration field ge„.,«e, an even, when ,he dutation of all olie„« of this 

T.meBaseNodehaved«enm„ed«,eirduraU„„.n,e value of duration field is the same as the 
Client with the longest duration. 

Line 5 lists the isActive field, which generates a true event when this node becomes 
active and a false event when this node becomes inactive. 

IntervalCue 

The following code portion illustrates the IntervalCue node. A description of the 
fields in the node follows thereafter. 

1) IntervalCue : CueNode { 

2) field Float period 1 

3) field Bool rampUp true 

4) field Float fraction 0 

5) field Bool isActive false 

} 

This object sends fraction events from 0 to 1 (or 1 to 0 if rampUp is false) as time 
advances. 

Line 2 of the code snippet lists the period field, which deterinines the time, in seconds, 
over which the fraction ramp advances. 

In line 3, if the rampUp field is true (the default) the fraction goes up ftx,m 0 to I over 
the duration of the IntervalCue. If false, the fraction goes down f«)m 1 to 0. If mediaTime is 
running backwards (when the rate is less than zero), the fraction goes down from 1 to 0 when 
rampUp field is tnae. and the fraction goes up f«,m O to 1 when the rampUp field is false. 
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In line 4, the fraction field sends an event with each call to evaluate() while this node 
.s active. If mediaTime is moving fonvard, fraction starts to output when this node fires and 
stops when this nodes reaches its firing time plus period. The value of fraction is described 



fraction = (mediaTime - firing time) * period 

Eqn. (2) 

Line 5 lists the isActive field, which sends a true event when the node becomes active 
and false when the node becomes inactive. If mediaTime is moving forward, the node 
becomes active when mediaTime becomes greater than or equal to firing time. This node 
becomes inactive when mediaTime becomes greater than or equal to firing time plus period 
If mediaTime is moving backward, the node becomes active when mediaTime becomes less 
than or equal to firing time plus period and inactive when mediaTime becomes less than or 
equal to firing time. The firing of these events is affected by the direction field. 
FieldCue 

The following code portion illustrates the FieldCue node. A description of the fields 
m the node follows thereafter. 

1) FieldCue : CueNode { 

2) field Field cueValue NULL 
^ 3) field Reld cueOut NULL 

This object sends cueValue as an event to cueOut when HeldCue fites ReldCue 

allows anyfield type to be s«»d emitted 11,e cueOut ™i„e can be touted toafieldof any 
type. UndeHned results can occurif d» cunen, typ. of e„.vriue is no. compaUble with the 
type of the destination field. 

In line 2 of the code portion, the cueValue field is the authored value that will be 
emitted when this riode fires. 

Line 3 lists the cueOut field, which sends an event with the value of cueValue when 
this node fires. 
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TimeCue 

The following code portion illustrates the TimeCue node. A description of the field ir 
the node follows thereafter. 

1) TimeCue : CueNode { 

2) field Time cueTime 0 

} 

This object sends the current wall clock time as an event to cueTime when TimeCue 

fires. 

Line 2 of the code portion lists the cueTime field, which sends an event with the 
current wall clock time when this node fires. 

The scoring constnict within the context of real-time scene composition enables the 
author to declaratively describe temporal control over a wide range of presentation and 
playback techniques, including: image flipbooks and image composite animations (e g 
animated GIF); video and audio clips and streams; geometric animation clips and streams, 
such as joint transformations, geometry morphs. and texture coordinates; animation of 
rendering parameters, such as lighting, fog. and transparency; modulation of parameters for 
behaviors, simulations, or generative systems; and dynamic control of asset loading, event 
routing, and logic functions. Fbr instance, the following example emits a string to pre-load an 
image asset, then perfomis an animation using that image, then runs a movie. The string in 
the following example can also be nin in leveise (i.e.. first the movie plays backwards tiien 
the animation plays backward and then the image disappears). 

1) Scone { 

2) timeBaseDEFTBTimeBase{ } • 

3) cue[ 

4) Fieldcue { 

5) cueValue String " " 

6) cueOut TO ISURF.URL 

7) direction -1 

8) } 

9) FieldCue { 

10) cueValue String "imagel .png" 

1 1) cueOut TO ISURF.url 

12) direction -10 
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13) } 

14) IntervalCue { 

15) delay 0.5 

16) period 2.5 # 2.5 second animation 

17) fraction TO Pl.fraction 

18) } 

19) DEFMCMediaCue { 

20) offset 2 

21) } 

22) Fieldcue { 

23) cueValue String " " 

24) cueOut TO ISURF.URL 

25) direction 1 

26) delay -0.5 

27) } 

28) Fieldcue { 

29) cueValue String "imagel .png" 

30) cueOut TO ISURF.URL 

31) direction -1 

32) delay -0.5 

33) } 

34) ] 

35) } 

36) # Slide out image 

37) DEFT Transform { 

38) children Shape { 

39) appearance Appearance { 

40) texture Texture { 

surface DEF KURF ImageSurface { 

43) } 

44) geometry IndexedFaceSet { ... } 

45) } 

46) } 

47) DEF PI Positionlnterpolator { 

48) key... 

49) keyValue ... 

50) value TO T.translation 

51) } 

52) # Movie 

53) Shape { 

54) appearance Appearance { 

55) texture Texture { 

56) surface MovicSurfacc { 

57) url "myMovie.mpg" 
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58) timeBase USE MC 

59) } 

60) } 

61) } 

62) geometry IndexedFaceSet { ... ) 

63) } 

All Cue nodes in a Score fire relative to the media time of the TimeBase. providing a 
conunon reference and thereby resulting in an accurate relationship between timing of various 
media assets. In the code snippet above, the ReldCue (line 9) fires as soon as the TimeBase 
staits because this FieldCue has default offset and delay fields thereby making the image 
appear. lines 35-45 of the code portion loads the iinage (200. Fig. 2A) on a surface. The 
IntervalCue (line 13) then starts 0.5 seconds later and runs for the next 2.5 seconds, increasing 
its fraction output from 0 to 1, The firing of the IntervalCue starts the animation (202, Fig. 
2A) of the image. Lines 46-50 control die animation. The MediaCue (line 18) staits 2 
seconds after the TimeBase starts, or when the IntervalCue is 1.5 seconds into its animation 
thereby starting the movie. Lines 51-62 loads the first frame (204, Fig. 2A) of the movie on 
the surface. When this string is played backwards, first the movie plays in reverse. Then 0.5 
seconds later the image appears, and 0.5 seconds after the image appears the animation staits. 
Animation is played in reverse for 2.5 seconds, when it stops and 0.5 seconds after that the 
image disappears. This example shows die ability of the Cues to be offset from each other or 
ftom the TimeBase and shows tiiat a subsequent Cue can start before the last one has finished. 

The MediaCue gives a synchronization tool to the autiior. A MediaCue is a form of 
Cue. which behaves exacfly like a TimeBase. In fact, a MediaCue can be used where a 
TimeBase can. as shown in the above example. But since a MediaCue is embedded in a timed 
sequence of events, an implementation has enough informatioii to request pre-loading on an 
asset. Fig. 2B illustrates synchronization of the mediasequence of Fig. 2A requiring pre- 
loading. For instance, in the above example, if the implementation knows tiiat a movie takes 
0.5 seconds to pre load and play instantly, after waiting (block 210) 1.5 seconds after the start 
of the TimeBase, in block 215, a "get ready" signal is sent to the MovieSurface. Upon receipt 
of get ready signal, in block 220 the movie is pre-loaded. This would give it the required 0.5 
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seconds to pre-load In block 225 a request to start is received, and upon receipt of the 
request to start, block 230 starts the movie instantly. 

The combination of the "HmieBase and media sequencing capabilities allowed in 
Blendo makes it possible to create presentations with complex timing. Fig. 3 shows time 
relationships of various components of a Blendo presentation. A viewer, upon selecting news 
presentation (360), sees a screen wherein he can select a story (362). Upon the user selecting 
story S3 from a choice of five stories SI, S2. S3. S4 and S5. a welcome screen with an 
announcer is displayed (364), On the welcome screen the viewer can choose to switch to 
another story (374) thereby discontinuing story S3. After the welcome statement, the screen 
transitions to the site of the story (366) and tiie selected story is played (368). At this point, 
the viewer can go to the next story, the previous story, rewind the present story or select to 
play an extended version of story (370) S3 or jump to (372), for example, another story S5. 
After the selected story is played the user can make the next selection. 

It is to be understood that the present invention is independent of Blendo, and it can be 
15 part of an embodiment separate from Blendo. It is also to be understood that the present 
invention is equally applicable to 2D scene rendering and 3D scene rendering. 

While particular embodiments of the present invention have been described it will be 
apparent to those skilled in the art that changes and modifications may be made without 
departing from this invention in its broader aspea and. therefore, the appended claims are to 
20 encompass within their scope all such changes and modifications as fall within the tnie sprit 
and scope of this invention. 
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